Method and system to control a workflow and method and system for providing a set of task-specific control parameters

ABSTRACT

The invention relates to a system and method to control a workflow comprising at least one task to be performed by a person (P3), wherein information is provided about at least one certain object (20, 32, 36, 38) related to the at least one task of the workflow, eye data (24, 26) are captured of at least one eye of the person (P3), and in dependency of the eye data (24, 26) and the information about the at least one certain object (20, 32, 36, 38) it is checked whether at least one task condition consisting in whether the task had been performed and/or whether the task is allowed to be performed is fulfilled. The invention also relates to a system and method for providing a set of task-specific control parameters (CP).

The invention relates to a system and method to control a workflowcomprising at least one task to be performed by a person, whereininformation about at least one certain object related to the at leastone task of the workflow is provided and eye data of at least one eye ofthe person are captured. The invention also relates to a system andmethod for providing a set of task-specific control parameters,according to which a workflow comprising at least one task to beperformed by a user is controlled.

From the prior art gaze-triggered events in various contexts are known.If for example a user fixates a predefined object, then a certain actionis triggered. This mechanism, however, is not embedded into a processthat aims to achieve an overarching goal of the process with the relatedoptions and consequences.

Therefore, it is an object of the present invention to provide animproved system and method to control a workflow. This object is solvedby a system and method with the features according to the independentclaims. Advantageous embodiments of the invention are presented in thedependent claims.

According to the method to control a workflow according to theinvention, the workflow comprises at least one task to be performed by aperson, wherein information about at least one certain object related tothe at least one task of the workflow is provided and eye data of atleast one eye of the person are captured. Moreover, in dependency of theeye data and the information about the at least one certain object, itis checked whether at least one task condition consisting in whether thetask had been performed and/or whether the task is allowed to beperformed is fulfilled.

Advantageously, eye data of the person can be used, especially withregard to the at least one certain object to check and monitor whethersingle tasks of a workflow have been performed, especially correctly, oreven if certain security conditions are fulfilled so that the person isallowed to perform the certain task. Therefore, the invention provides acontrol mechanism, which can enhance the quality of the outcome of aworkflow and even reduce the risks for dangerous situations.

For example, the task can only be allowed to be performed by the person,if the person has seen and optionally acknowledged the criticalinformation displayed, which can be derived from an analysis of thecaptured eye data. According to another example, if the task to beperformed by the person, like a helicopter pilot, consists in scanningall instruments regularly in certain time intervals, then this can bechecked on the basis of the captured eye data as well.

Advantageously, gaze data are captured as the eye data, especially agaze direction and/or a point of the regard of the person, in particularin relation to his/her environment, especially relative to the at leastone certain object. On the basis of a captured gaze direction or pointof regard it can easily be checked whether the person has looked at thecertain object or not, or in general on objects which are relevant forperforming the task. Therefore, the use of gaze data can greatly improvea control of a workflow. Moreover, the captured eye data can also be amotion pattern relating to a motion of the at least one eye of theperson and/or of at least one eye feature of the at least one eye of theperson, like a pupil and/or an eyelid. On the basis of such eye motionpatterns, like one or more fixations of the eye or uncoordinated eyemovements, and eye feature motion like eye lid closures or the rate ofeye lid closure, so called PERCLOS, and motion of the pupil like pupilcontraction, advantageously much more useful information about theperson can be provided, like explained in the following.

According to another advantageous embodiment of the invention a state ofthe person, especially an emotional state and/or a state of attentionand/or a state of mind and/or a fit for duty state, is determined independency of the captured eye data, wherein the checking whether the atleast one task condition is fulfilled is performed in dependency of thedetermined state. On the basis of above named motion patterns related tothe motion of the eye or parts of the eye like the pupil or the eye lid,a current state of the user can be determined, which can giveinformation about e.g. whether a user is attentive or concentrated whenlooking at an object or not, whether the user recognizes an object oronly is looking at the object unintentionally or accidentally. Also afit for duty state, which can be determined by fit for dutymeasurements, e.g. based on pupil measurements and/or eye lid closuremeasurements, it can be determined whether the person is tired orunfocused or not. Also the determining of the emotional state or thestate of mind can be based on pupil measurements. Advantageously alsostates of the person can be considered when checking whether the personcorrectly performs a task, like when actively and concentrated lookingat objects relevant for the task, or whether the person is allowed toperform the task, like when being not tired, nervous, afraid ordistracted.

According to another advantageous embodiment of the invention it isclassified whether the task condition is fulfilled or not by comparing aposition information derived from the eye data, especially about thepoint of regard of the person, with a position information about the atleast one certain object. On the basis of such a comparison muchinformation about the person performing the task can be derived, forexample whether the person has looked at the relevant objects forperforming the task, if the person has seen the relevant information forperforming a task, or to be allowed to perform the task, if the personhas checked relevant objects in a predefined order, or within certaintime intervals, and so on. Therefore, much information can be derivedabout whether the person is performing the task correctly or not orwhether a person has gathered enough information to be allowed toperform the task.

Therefore, it is an advantageous embodiment of the invention that it isclassified whether the task condition is fulfilled or not according toat least one classification condition consisting in that the taskcondition is classified as to be fulfilled, if it is detected on thebasis of the eye data that the person has looked at and/or acknowledgedthe at least one certain object and/or has looked at and/or acknowledgeda plurality of certain objects, one of which is the at least one certainobject, and/or had looked at and/or acknowledged a plurality of certainobjects, one of which is the at least one certain object, in apredefined timely order, and/or had looked at and/or acknowledged the atleast one certain object within a predefined time interval, or regularlywithin predefined time intervals.

These classification conditions are very suitable for deciding, whethera task has been performed correctly or if the person has undertaken thenecessary steps or read the necessary information to be allowed toperform the task. Furthermore, whether the person has not only looked atan object but also has acknowledged this object when looking at it, canbe determined on the basis of above named motion pattern.

According to another embodiment of the invention it is classifiedwhether the task condition is fulfilled or not according to at least onesecond classification condition consisting in that the task condition isclassified as to be fulfilled if it is detected on the basis of the eyedata and a comparison of the eye data with stored eye data that theperson is an authorized person for performing the task. Thereforeadvantageously the person can be identified on the basis of the capturedeye data, which can be checked against a data base to determine if theperson, e.g. a worker, has a license or a required skill level or otherkind of authorization e.g. to operate a chainsaw, or in general toperform the task.

Moreover, as already mentioned, further criterial relating to the stateof the person, like a state of mind or a state of attention can beapplied for classifying whether the task condition is fulfilled or not.Therefore, according to another advantageous embodiment of theinvention, it is classified whether the task condition is fulfilled ornot according to at least one classification condition consisting inthat the task condition is classified as to be fulfilled, if it isdetected in dependency of the determined state of the person that theperson is attentive, especially when looking at the at least one certainobject and/or at a plurality of certain objects, one of which is the atleast one certain object, and/or at a plurality of certain objects, oneof which is the at least one certain object, in a predefined timelyorder, and/or at the at least one certain object within a predefinedtime interval, or regularly within predefined time intervals. E.g. whenthe eye performs one or more fixations on the certain object it can bedetermined that the person is attentive when looking at the certainobject. Also when it is determined, e.g. on the basis of eye lid closurepatterns, that the person is not tired when looking at the certainobject, it can be determined that the person is attentive when lookingat the certain object.

In particular, the at least one classification condition is specified independency of the kind of the task to be performed by the person.Advantageously, suitable classification conditions can be chosen independency of the task to be performed by the person. This way, for eachtask of a workflow or also of different workflows one or more ofabove-named classification conditions can be specified, and maybe savedto a storage device, and which are suitable for assessing whether a taskhad been performed correctly and/or whether the task should be allowedto be performed or not and/or how the user did the task and/or which eyedata the user produced while doing the task.

According to an advantageous embodiment of the invention the at leastone certain object is an object in the environment of the person. Forexample, this certain object may be an object the task has to beperformed with, like a device or work tool. The certain object in theenvironment can also be an object that provides information to theperson and, at which the person has to look to perform the taskcorrectly, like the display of a luggage scanner or also a certaindisplay part of the display, instruments in the cockpit of an aircraftor instruments on a control panel, a medical image like a mammographicx-ray image, and so on. Therefore, many tasks involving certain objectscan be assessed according to whether they have been performed by theperson or not.

According to another advantageous embodiment of the invention images ofthe environment of the person are captured and the at least one certainobject is identified in at least one of the images, especially whereinthe information about a position of the at least one certain object fromthe at least one image is derived. The images can be captured forexample by means of a camera, preferably a head-mounted camera, whichcan capture images of the environment of a person while the person isperforming the task. The captured images can be processed and analyzedfor the purpose of checking whether the person is looking at one or morepredefined certain objects or not. If for example the certain objectcannot be identified in one of the captured images, e.g. by means ofknown object recognition methods or algorithms, while the person isperforming the task, then it can be concluded that the person had notperformed the task correctly. Moreover, if the at least one certainobject is identified in one of the images, the position of the objectcan be compared with a gaze direction of the person to check whether theperson is looking at the certain object or not when performing the task.If not, again it can be concluded that the person may not performed thetask correctly. Moreover, the image of the environment may also be usedto map a user's gaze data to a specific location and to aggregate itover the time a user is performing a task e.g. if a user is performing avisual inspection of a part, the users gaze mapped to the object in apreviously captured image, aggregated over the checking period may bestored as documentation of such task executed by such user.

A further great advantage is, that the system for performing the methodaccording to the invention and its embodiments can be provided as ahead-mounted device, for example having a scene camera for capturing theimages of the environment of the person, as well as an eye-tracker fortracking the gaze direction of the person. Such a head-mounted systemprovides a lot of flexibility as it can be used in every kind ofenvironment and for every kind of task. However, the system can also beprovided as a remote system not worn by the person or attached to beperson, as well. This can be very advantageous if the at least onecertain object or other certain objects relating to the task are fixedin their position, like instruments of a cockpit, so that the gazedirection of a person can be captured by means of a remote eye-trackerin the position of the certain object are known to the system so thatthe system only has to compare the current point of regard of the userwith the positions of the predefined certain objects to be able toassess whether the user looks at those objects or not.

According to another advantageous embodiment of the invention a displayof a task information about the at least one task to be performed isdisplayed, especially by means of a display device, wherein the displayof the task information is the at least one certain object. This way itcan advantageously be checked whether a person has read or seen theinformation relating to the task to be performed in advance ofperforming the task so that on the basis of this information it can beassessed whether the task is allowed to be performed or not.

Therefore, in dependency of the captured gaze data it can be checkedwhether the person has seen and/or read task information, especiallyagain by comparing the position of the gaze point of the person with aposition of the displayed task information. So the person might only beallowed to perform the task if the person has read or seen the taskinformation. By this embodiment a high degree of safety can beestablished when controlling the workflow. Also guidance for the personcan be provided by means of the task information telling how the taskhas to be performed. Such task information can be displayed on a normaldisplay, like a computer display or even by means of a head-mounteddisplay.

Preferably, the task information is a warning information and/orinformation or instruction about how to perform the task and/orinformation specifying the task to be performed and/or an opticalindication of an object, which is relevant for performing the task inthe environment of the person. A warning information can for exampleprovide safety instructions, the person has to read before he is allowedto perform the task. Task information can also specify the task to beperformed or also how to perform the task, like a description of singlesteps to be performed by the person, and again only if the person hasread these instructions or noticed/acknowledged the information then heis allowed to perform the task. Also objects in the environment of theperson can be optically marked by means for the displayed taskinformation, e.g. displayed by means of an augmented reality device likeaugmented reality glasses. Also this can give guidance to the person andmay force the person to notice objects in the environment, which areimportant for performing the task correctly. If for example the personhas to tighten several bolts in a predefined order, the task informationcan optically mark, e. g. by means of a head-mounted display, oraugmented reality glasses, the first bolt, after that the second one,and so on, to show the person in what order the bolts have to betightened. At the same time, the system can check, whether the personhas performed the task correctly. If the person has looked at all ofthese bolts in the correct order, then it can be assumed, that the taskwas performed correctly, and otherwise, if for example the person forgotto look at one of the bolts, it can be assumed that the task was notperformed correctly. In case of a visual inspection task were the orderof checking operations may not be relevant the invention is veryadvantageous by providing guidance to the user e.g. by visually markingall still to be checked components and removing such marking after thesystem detected a check e.g. when the user gazed at such part.

By means of such task information it is possible to provide safetyinformation on the one hand as well as guidance on the other hand. Thisway, augmented reality systems could also be advantageously used to helpuntrained personal to do tasks which used to require a professional,with enforcing certain sequences and/or coverage of visual intake. Thiscan be realized safely and with less liability.

According to another advantageous embodiment of the invention, if thetask condition is not fulfilled, as at least one first consequence

-   -   a subsequent task of the workflow is prohibited or blocked to be        performed; and/or    -   the at least one task is not allowed to be performed; and/or    -   information about the task condition being not fulfilled is        stored and/or transmitted, e.g. to another device like that of a        supervisor; and/or    -   a warning information is outputted, especially to the person;        and/or    -   a device for performing the at least one task or a function of        the device is blocked or disabled; and/or    -   a device for performing a subsequent task of the workflow or a        function of the device is blocked or disabled; and/or    -   an output of a result of the at least one task of the workflow        is blocked.

These first consequences might again depend on the task to be performed.In general, it is very advantageous to output a warning information tothe person so that the person can be informed that the task he/she hadto perform was not performed correctly or completely or that conditionsthat have to be fulfilled for performing a task, like reading the safetyinstructions, are not fulfilled. Also it is very advantageous to storeinformation about the task condition being not fulfilled, which highlyfacilitates error tracing. Moreover, if the task condition consists inwhether the task is allowed to be performed or not, it is advantageousthat the at least one task is not allowed to be performed if the taskcondition is not fulfilled. This can be done for example if the systemfor performing the method according to the invention or its embodimentsas coupled to a device for performing the task, in this case, a functionof the device itself can be blocked or disabled so that the task cannotbe performed. Even if the person already had performed the task, but didnot do this correctly, in the same way a subsequent task of the workflowcan be prohibited or blocked to be performed.

Similarly, according to another embodiment of the invention, if the taskcondition is fulfilled, as at least one second consequence

-   -   a performing of the at least one task is allowed; and/or    -   a performing of a subsequent task of the workflow is allowed;        and/or    -   a positive feedback information is outputted to the person;        and/or a device for performing the at least one task or a        function of the device is enabled; and/or    -   a device for performing a subsequent task of the workflow or a        function of the device is enabled; and/or    -   a result of the at least one task or of the workflow is        outputted; and/or    -   information about the task condition being fulfilled is stored        and/or transmitted.

This way advantageously the person again can be notified about thesuccessful performance of the task, this information again can be storedand also be used as control information for controlling devices withwhich the task or a subsequent task has to be performed. Which of thesefirst or second consequences is applied for a specific task againdepends on the kind of task to be performed.

Therefore, it is an advantageous embodiment of the invention that forthe at least one task, especially for each task of the workflow, atask-specific set of rules is specified, which define at least one of atleast one classification condition, according to which it is classifiedwhether the task condition is fulfilled or not, especially which is oneor more of the above-named classification conditions, the at least onecertain object, the position of the at least one certain object, thetask condition, a first consequence of the task condition being notfulfilled, especially at least one of the first consequences namedabove, and the second consequence of the task condition being fulfilled,especially which is at least one of the second consequences named above.This way, for each task a suitable set of rules can be defined, whichare then used to control the workflow, to make decision when a task isperformed, performed correctly, what and which premises are made to beallowed to perform a task, and which consequences a correct or incorrectperformance of a task may have.

Moreover, data relating to test persons, e.g. trained persons orprofessionals, performing the at least one task are collected during thetest persons are performing the at least one task and the task specificset of rules is learned or derived from the collected data. Thisadvantageous automatic learning process is explained in more detail withregard to the second aspect of this invention.

Furthermore, according to another advantageous embodiment of theinvention, the eye data, in particular which are captured while theperson is performing the at least one task, are stored, preferably alsotogether with respective information about the at least one certainobject, like in form of a video stream of images of the environment ofthe person performing the task, wherein each image contains the point ofregard of the person for that point in time at which the respectiveimage of the environment was captured. By storing these eye data orinformation derived from these eye data, like gaze pattern, or attentionstates, the whole process of preforming the task or several tasks or thewhole workflow can documented, which can serve for liability purposes aswell as training material for other users or for a system which canderive TC e.g. via machine learning by studying user behavior whileperforming a task. If the task e.g. consists in inspecting a certaincomponent or device and a costumer owning that device then notices afailure of the device and claims that the device wasn't inspectedcorrectly, by means of the stored eye or gaze data prove can be providedthat all parts of the device have been inspected correctly and that thefailure of the device is not due to a failure of inspection.

Therefore, by means of the invention and its embodiments guidelines canbe established in order to ensure high quality, and especially indangerous situations and dangerous processes to protect the worker oruser. The system helps to enforce appropriate visual intake of theperson to avoid critical omissions in perception or decision-making.Also, single tasks during the performance of the workflow or theperformance of a whole process can be monitored individually and notonly results of such a process. Often, it remains unknown which (visual)input has been missed by the person leading to an unstable process or ifthe intended result is not achieved the cause remains hidden. This iseven more critical, when the result can be observed only a long timelater or when it is too late or has become very costly to correct. Thisis typical e. g. in medical screening and diagnostic procedures or inpreventive quality inspection task. In these cases the process itselfensures the quality as the result and their correctness can only beassessed much later, often only after irreversible damage has occurred,which should have been prevented by the diagnostic or inspectionprocedure in the first place. Instead, the invention manages to checksingle tasks and steps of a workflow with regard to their correctness sothat misconduct or negligence of the person can be detected or evenavoided.

The invention also relates to a system to control a workflow comprisingat least one task to be performed by a person, wherein the systemcomprises a device for providing information about at least one certainobject related to the at least one task of the workflow and a capturingdevice for capturing eye data of at least one eye of the person.Moreover, the system comprises a processing unit, which is configured tocheck whether at least one task condition consisting in whether the taskhad been performed and/or whether the task is allowed to be performed isfulfilled in dependency of the eye data and the information about the atleast one object.

The preferred embodiments and advantages thereof described with regardto the method to control a workflow according to the inventioncorrespondingly apply to the system according to the invention, whereinin particular the embodiments of the method according to the inventionconstitute further preferred embodiments of the system according to theinvention. Especially, the system according to the invention isconfigured to perform the method to control a workflow according to theinvention and/or its preferred embodiments.

Moreover, the device for providing information about the at least onecertain object can be for example a camera, like a scene camera of ahead-mounted device. It could also be a storage device, in which objectinformation about the at least one certain object is stored or loaded,especially the kind and/or position of these objects. The capturingdevice for capturing eye data can be an eye-tracking device, like aremote eye tracker or also a head-mounted eye tracker, e. g. integratedinto an augmented reality head-mounted device.

According to another aspect the invention relates to a method forproviding a set of task-specific control parameters according to which aworkflow comprising at least one task to be performed by a user iscontrolled. According to this method eye data of at least one eye of atest person are captured during the test person is performing the task.Furthermore, information about at least one visible object in theenvironment of the test person is provided, in which the test person isperforming the task. Moreover, from the information about at least onevisible object and the eye data information about the behavior of thetest person performing the task with regard to the environment of thetest person is derived and therefrom task-specific control parametersare derived, according to which the workflow is controlled for the user,wherein the task-specific control parameters at least specify a taskcondition, which specifies whether the task had been performed and/orwhether the task is allowed to be performed.

This method has the great advantage that the set of task-specificcontrol parameters, which especially can also be used for and combinedwith the method and system to control a workflow according to the firstaspect of the invention and its embodiments, can be taught to a systemautomatically without having to specify these parameters manually forevery single task of a workflow or also the tasks of differentworkflows. This way one or more test persons, like a professional, whoknows how to perform the task correctly or who knows which are thenecessary steps to perform the task safely, can be observed or monitoredtogether with his/her environment when performing the task, and fromthis observation the task-specific control parameters can be derived tospecify the task condition. So from the behavior of the test person withregard to its environment information can be derived for example likewhich are the relevant objects for performing the task, which is thenecessary information the test person captures with his eyes, whether itis necessary to perform certain steps within time limits, or whether thetimely order of performing single steps is relevant or not. All theseinformation can advantageously be derived by comparing the gazedirection, especially the point of regard, of the test person withrespect to its environment or be added manually as constraints or belearned from other parameters or be learned by how different peopleperform the task.

In particular, these task-specific control parameters correspond to thetask-specific set or rules explained with regard to the first aspect ofthe invention. Therefore, according to an advantageous embodiment of theinvention the task-specific control parameters specify additionally oralternatively at least one of at least one classification condition,according to which it is classified whether the task condition isfulfilled or not, and/or at least one certain object in the environment,which is classified as being a relevant object for performing the task,and/or a first consequence of the task condition being not fulfilled,and/or a second consequence of the task condition being fulfilled.Preferably, the classification condition, the certain object, the firstand second consequences relate to the corresponding ones alreadyexplained with regard to the first aspect of the invention. All theseparameters can advantageously be derived from the observation of thetest person and its environment. Also the task-specific controlparameters may specify at least one of above-named classificationconditions, the certain object, first and second consequences as analternative of specifying the task condition. Or in other words, theseinformation can all be derived from the observation of the test personand its behavior with regard to its environment when performing the taskindependently from each other. Moreover, to specify the classificationcondition, even more useful information can be derived from the behaviorof the test person with regard to the environment, like the alreadymentioned timely order of performing single tasks or steps of a task andwhether such a timely order is relevant or not, as well as single tasksor steps have to be performed within certain time limits, regularly ornot.

Furthermore, the method for providing a set of task-specific controlparameters according to the second aspect of the invention and itsembodiments are especially advantageous when performed on a system,which is configured as head-mounted device, e. g. with a head-mountedscene camera, for capturing images of an environment for providinginformation about at least an object in the environment and with anintegrated eye-tracking device for capturing the eye data of the testperson. By means of such a head-mounted device, it is possible that aprofessional simply wears this device during the performance of the atleast one task wherein this device then derives from the capturedinformation the set of task-specific control parameters, which can thenbe used to control the workflow comprising this task for every otheruser subsequently having to perform this task.

According to an advantageous embodiment the captured eye data areinterpreted with regard to their relevance for the task, especiallywherein a state of the person, in particular an emotional state and/or astate of attention and/or a state of mind and/or a fit for duty state,is determined in dependency of the captured eye data and used forinterpreting the captured eye data. Also the task specific controlparameters can be determined in dependency of the determined state ofthe test person. E.g. if the test person is looking at an object, butonly shortly or unfocused, then therefrom the information can be derivedthat this object probably is not important for performing the task.

According to another advantageous embodiment of the invention, forproviding the information about the at least one visible object in theenvironment of the test person images of the environment of the testperson are captured while the test person is performing the task.

Moreover, it is preferred that the capturing of the eye data of the atleast one eye of the test person as well as the providing of informationabout the at least one visible object in the environment of the testperson are performed a plurality of times and the set of task-specificcontrol parameters is then derived from the information about theenvironment provided the plurality of times and the eye data capturedthe plurality of times. This way the test person, like a professional,can be monitored, when performing the task a plurality of times, likehundreds or even thousands of times, and then these data can be used asdescribed before for deriving the set of task-specific controlparameters. This procedure has the great advantage that the accuracy ofthis method for providing the task-specific control parameters can begreatly enhanced. This way for example situations in which the testperson looks at an object unintentionally and situations in which thetest person looks at objects intentionally for performing the task canbe distinguished with higher reliability. All these situations, in whichthe timely order of performing tasks or single steps, play an importantrole for performing the task and in which not, can be distinguished moreeasily. For example, if the test person is looking at several predefinedobjects in predefined manner, when performing the task, one cannotconclude with high reliability that this order is important forperforming the task. If, however, the test person is monitored severaltimes when performing the task, each time the test person is looking atthese certain objects in the same timely order, then one can concludewith high reliability that the order is important. On the other hand, ifthe test person is monitored when performing the same task for severaltimes, and each time the test person is looking at these certainobjects, but not in the same timely order, then one can conclude thatthese objects are relevant objects, however, the order is not importantfor performing the task. Same applies for deriving information abouttime limits. Therefore, by repeating these steps several times one cangather lots of information about what are the important objects forperforming a task, is the timely order of performing single stepsrelevant or not, are there time limits for performing single steps ofthe task, or the task itself, and so on with high reliability.

The repetition of the capturing and the providing a plurality of timesdoes not necessarily have to relate to the same test person. Alsodifferent test persons can be observed and monitored when performing atask or optionally each test person can be observed or monitored aplurality of times, and then all these data, namely the eye data and thedata relating to the respective environment and objects therein can beused for deriving the set of task-specific control parameters. This wayalso personal preferences of the test persons can be eliminated byaveraging over many test persons.

According to another advantageous embodiment of the invention, thecapturing of the eye data and the providing of information about the atleast one visible object are performed a plurality of times fordifferent test persons, each comprising a predefined skill level,wherein each time a result of the performance of the at least one taskis assigned to a quality level, wherein the set of task-specific controlparameters are additionally derived in dependency of the skill levelsand quality levels. The skill levels and/or quality levels can be ratedmanually. The quality levels can also be rated automatically accordingto a suitable rating metric. The great advantage of this embodiment ofthe invention is that also the skill levels of the test personsperforming a task as well as additionally or alternatively the qualitiesof the outcomes or results can be taken into consideration when derivingthe set of task-specific control parameters. For example, if the testperson has a high skill level, then the probability that this testperson performs the task correctly is also high. Information gatheredfrom the monitoring of this test person when performing the task can beassumed to be correct with high probability. However, if a person withlow skill level performs the task, then the probability that not eachstep of the task or the task itself is performed correctly, can beassumed to be higher. So for example when deriving the set oftask-specific control parameters from these pluralities of data sets ofdifferent persons, the data sets relating to test persons with higherskill levels can be weighted stronger with regard to their correctness.The same applies for quality levels assigned to the results of theperformance of the task. If result of a performance of the taskperformed has a low quality, then it can be assumed that the task or atleast some steps thereof were not performed correctly. If on the otherhand such a result has a high quality, then it can be assumed that thetasks or all steps thereof were performed correctly, at least with highprobability. Also here corresponding weights can be used for the resultsof different quality levels, when deriving the set of task-specificcontrol parameters.

According to another advantageous embodiment of the invention thecapturing of the eye data and the providing of information about the atleast one visible object are performed a plurality of times fordifferent test persons, wherein each test person comprising a predefinedstate, especially a state of mind and/or an emotional state and or astate of attention, wherein in the set of task-specific controlparameters is additionally derived in dependency of the predefinedstate. Therefore also the states of the test persons can additionally betaken into account and useful information can be derive therefrom.

According to another advantageous embodiment of the invention forderiving the set of task-specific control parameters an adaptive methodis applied, especially machine learning, a neural network training, astatistical method or functional analytics. Such adaptive methods areespecially advantageous when capturing data from different test personsor plurality of times for deriving the set of task-specific controlparameters.

According to the second aspect of the invention it is advantageouslypossible to determine the set of rules and the relevant scene data basedon analysis of a number of performances of the intended procedure byusers with all skill levels and performance outcomes. This can be doneanalytically, statistically, or via machine learning approaches, usingeye tracker data, scene video data and process result and performancedata as inputs to determine the scene data and the set of rules relevantto be observed and controlled for, in order to achieve intended processresults and performance.

This second aspect of the invention also relates to a system forproviding a set of task-specific control parameters according to which aworkflow comprising at least one task to be performed by a user iscontrolled. The system comprises a capturing device, like an eyetracker, for capturing eye data of at least one eye of a test personwhile the test person is performing the task. Furthermore, the systemcomprises a device, like a scene camera, for providing information aboutat least one visible object in the environment of the test person, inwhich the test person is performing the task. Moreover, the systemcomprises a processing unit, e.g. a micro-controller, configured toderive from the information about the at least one object and the eyedata information about the behavior of the test person performing thetask with regard to the environment of the test person and to derivetherefrom task-specific control parameters, according to which theworkflow comprising the at least one task is controlled for the user,wherein the task-specific control parameters at least specify a taskcondition, which specifies whether the task had been performed and/orwhether the task is allowed to be performed.

The advantages described with regard to the method for providing a setof task-specific control parameters according to the second aspect ofthe invention and its embodiments correspondingly apply to the systemfor providing a set of task-specific control parameters according to thesecond aspect of the invention. Moreover, the features and preferredembodiments described with regard to the method providing a set oftask-specific control parameters according to the second aspect of theinvention constitute further preferred embodiments of the system forproviding a set of task-specific control parameters according to thesecond aspect of the invention.

Furthermore, the preferred embodiments and features of the method andsystem to control a workflow according to the first aspect of theinvention can also be combined with the method and system for providinga set of task-specific control parameters according to the second aspectof the invention and its embodiments and features thereof.

Further features of the invention and advantages thereof derive from theclaims, the figures, and the description of the figures. All featuresand feature combinations previously mentioned in the description as wellas the features and feature combinations mentioned further along in thedescription of the figures and/or shown solely in the figures are notonly usable in the combination indicated in each place but also indifferent combinations or on their own. The invention is now explainedin more detail with reference to individual preferred embodiments andwith reference to the attached drawings.

These show in:

FIG. 1 a schematic illustration of a system and method for providing aset of task-specific control parameters, according to which a workflowcomprising at least one task to be performed by a user is controlled,according to an embodiment of the invention;

FIG. 2 a flowchart for illustrating a method to control a workflowaccording to an embodiment of the invention;

FIG. 3 a flowchart for illustrating a method to control a workflowaccording to another embodiment of the invention;

FIG. 4 a schematic illustration of a system to control a workflowaccording to an embodiment of the invention; and

FIG. 5 a schematic illustration of a system to control a workflowaccording to another embodiment of the invention.

FIG. 1 shows a system 10 a for performing a method for providing a setof task-specific control parameters CP, according to which a workflowcomprising at least one task to be performed by a user is controlled,according to an embodiment of the invention. The system 10 a maycomprise one or more head-mounted devices 12, each comprising a scenecamera 12 a and eye cameras 12 b as a part of respective eye trackersintegrated into the head-mounted devices 12. Furthermore, the system 10a comprises a processing unit 14, which can be provided with a neuralnetwork 14 a, to calculate the control parameters CP, which can bestored in a storage device 16 of the system 10 a.

To provide the set of task-specific control parameters CP one or moretest persons P1, P2 can wear the head-mounted devices 12 during they areperforming a specific task of a workflow. While these test persons areperforming this task, the respective scene cameras 12 a capture aplurality of scene images in form of a scene video of the environment ofthe test persons P1, P2, in which these test persons P1, P2 areperforming the task. On the basis of the captured scene imagesinformation about objects in the environment of the test persons P1, P2can be provided. These captured scene images can then be provided inform of scene data S1, S2 to the processing unit. At the same time,namely when the test persons P1, P2 are performing the task, the eyecameras 12 b capture eye data of the respective eyes of the test personsP1, P2, especially in the form of eye images, on the basis of which aneye tracker can calculate the respective gaze directions of the testpersons P1, P2. These gaze directions or the eye data E1, E2 in general,can also be provided to the processing unit 14. The captured eye dataE1, E2 can be set in relation to the respective scene data S1, S2 toderive for example information about at what point of the environment ofthe respective test person P1, P2 the test person P1, P2 was looking ata certain time, especially a respective gaze point with regard to theenvironment of the test person P1, P2 for each captured scene image canbe calculated.

The objective of this method is to find out for example, which objectsin the environment of a person P1, P2 are relevant objects forperforming the certain task, whether there is a relevant timely order ofsingle steps of the certain task, or whether single steps of a task haveto be performed within certain time intervals, whether certain stepshave to be repeated, especially more often than other steps, and so on.Now all this information can advantageously be derived from the scenedata S1, S2 and the eye data E1, E2. For this purpose it is veryadvantageous if the scene data S1, S2 and the eye data E1, E2 are notonly captured for one single performance of a certain task but also fora plurality of performances of one and the same certain task, either byone and the same test person P1, P2 or a plurality of different testpersons P1, P2, at the same time or subsequently.

Objects in the environment of the test persons P1, P2, which can beidentified for example on the basis of well known object detectionalgorithms, can be for example identified as relevant objects for thecertain tasks if these objects have been looked at a minimum number oftimes and/or for a certain minimum time duration and so on. Also objectscan be identified as relevant objects if the person P1, P2 was fixatingthem for a minimum time duration.

To verify such results, the results can be compared with each other. Iffor example an object was identified as a relevant object on the basisof the scene and eye data S1, E1 of a first person P1 and the sameobject was also identified as being a relevant object according to thescene and eye data S2, E2 of all other test persons P2, then this objectcan be classified to be a relevant object with high reliability. If thisobject, however, was identified only once to be a relevant object, thenit probably was looked at the minimum number of times or for the certainminimum time duration unintentionally.

The same applies to deriving information about whether the timely orderof performing single steps of the task is relevant or not. This forexample can be assessed on the basis of whether objects have been lookedat in a certain timely sequence for a minimum number of times for anumber of repetition of the task or number of test persons.

Also additional information can be used when determining the controlparameters CP, like information about the skill level SL of a respectivetest person P1, P2, as well as a quality level RQ of an outcome orresult of the task. The skill level SL of the respective test personsP1, P2 and/or the result qualities RQ can be rated and inputted into thesystem 10 a manually or rated by the system 10 a itself. This way forexample the input data of the respective test person P1, P2 can then beweighted according to their respective skill levels SL. Also respectiveweight can be applied depending on the result quality RQ. Also furtherinformation about the specific task can be derived like whether thereare correlations between the duration of looking at an object and abetter result, or looking at a certain object more often and a betterresult. Similarly, it can be determined whether performing certain stepsof the task within time limits is relevant or not, or whether timelimits or time intervals are important or not.

By means of an analysis of these data, namely the scene data S1, S2, theeye data E1, E2 and optionally respective skill levels SL and resultqualities RQ, the control parameters CP for a specific task can bederived and be stored in the storage device 16. These control parametersCP therefore can specify for example a task condition, which itselfspecifies whether this task had been performed and/or whether the taskis allowed to be performed, for example on the basis of theidentification of objects as being relevant for the task and othercriteria explained above.

These task-specific control parameters CP can then advantageously beused for controlling a workflow comprising this specific task to besubsequently performed by any other user.

Moreover, the processing unit 14 can also use other methods for derivingthe control parameters CP besides neural networks 14 a, like otheradaptive methods for example statistical methods or functionalanalytics.

FIG. 2 shows a flowchart for illustrating a method to control a workflowaccording to an embodiment of the invention. This workflow comprises atleast one task, for which it has been determined, for example by meansof a method described with regard to FIG. 1, that a certain number ofobjects in the environment of the person, who has to perform the task,are relevant, wherein furthermore these relevant objects can be assignedto single steps of the task in a timely order, which is relevant aswell. An example of such a task could be for example that a worker hasto tighten a certain number of screws in predefined order on aworkpiece. Another example could be a person has to make coffee with acoffee machine and has to fill in water into the water tank beforepressing the button for activation of the coffee machine.

The method starts in S10 with a start of the workflow control. Afterthat, information can be displayed to the person on any kind of displaydevice in S12 informing him/her that he has to take care of the timelyorder, in which single steps have to be performed. After that in S14 itcan be checked whether the person had read the information or not. Thiscan be done for example by comparing captured points of regard of theperson with the position, in which the information in S12 is displayed.If the person has not read the information, optionally a warninginformation can be displayed in S16, especially on any kind ofdisplaying device, and/or a working tool like an electric screw driveror a coffee machine in above-named examples, can be blocked or theactivation can be prohibited in S18. Moreover, as long as the person hasnot read the information the information about minding the order ofsteps keeps being displayed. If it is recognized in S14 that the personhad read the information, then the task is allowed to be performed inS20. Reading the displayed information therefore constitutes the taskcondition defining whether the task is allowed to be performed or not.So, if now the task is allowed to be performed, optionally a workingtool, like the above-named, can be enabled in S22 and moreoverguidelines can be provided for the person in S24 giving further adviceof how to perform the task. For example, all relevant objects relatingto the task can be optically marked or highlighted, for example by meansof augmented reality glasses, by means of which such additionalinformation can be overlaid over the respective environment of theperson. Also numbers can be displayed defining the order, in which theperson has to use the relevant objects. For example, if the person looksat the screws, these can be numbered by means of the augmented realitydevice to show in what order the person has to tighten them on theworkpiece. If the user looks at the coffee machine, also the water tankand the press button can be numbered correspondingly. The displaying ofthe numbers in their correct positions relative to the respectiveobjects of the environment can again be derived from scene images of ahead-mounted device in combination with object recognition algorithms.Then again in S26 gaze data of the user are additionally used to checkwhether the person has looked at all the relevant objects, especially inthe predefined order. So in case the person missed to look at a certainobject or looked at all of these objects but not in the correct order,optional consequences may be for example displaying a warning to theperson in S28 and storing the error data about having detected that thetask was not performed correctly, for example in a storage device in S30and/or blocking the output of a result of the task in S32. By displayinga warning the person can advantageously be informed that he has notperformed the task correctly, storing error data is very advantageousfor failure analysis and by blocking the output of a result on the onehand the person again can notice that something is wrong and he has notperformed the task correctly and on the other hand further negativeconsequences of the task being performed incorrectly can be avoided,like overheating the coffee machine when brewing coffee without havingwater in the water tank.

If, however in S26 it is determined that a person has looked at allrelevant objects in the predefined order, the task is considered to befulfilled in S29, meaning that a second task condition consisting inwhether the task had been performed is fulfilled. This procedure can beperformed again for every single task of a workflow, if a workflowcomprises several tasks. Also the next task of a workflow might only beallowed, if the previous task is considered to be fulfilled. So in thiscase, either in S30 the next task is allowed to be performed oralternatively the result of the task or the workflow is outputted.

FIG. 3 shows a flowchart for illustrating a method to control a workflowaccording to another embodiment of the invention. This method is at thebeginning identical to the method already explained with regard to FIG.2. However, in this case, after the task is considered to be allowed tobe performed in S20 and optional a working tool is enabled in S22, onlythe first relevant object of the plurality of relevant objects accordingto the predefined order is optically marked by display means, likeaugmented reality glasses in S32. After that in S34 it is checkedwhether the person has looked at the first object or not. If it isdetermined that the person has looked at the first object the sameprocedure can be repeated with regard to the second object, a thirdobject and so on until the last object, which is marked in S36. Again,in S38 it is checked whether the person has looked at the last object.If, however, it is determined in S34 or S38 or in any intermediate stepfor checking whether the person had looked at the object of the relevantnumber, that a person has not looked at the object, the respectiveobject can stay marked until the person has. Moreover, optionally againa warning can be displayed to the person informing him/her that the taskis not fulfilled yet in S40 and also optionally a device the person isworking with or has to perform the task with can be disabled in S42. Iffinally all these conditions are fulfilled, namely the person has lookedat each object in the predefined order task is again considered to befulfilled in S29 and the next task may be allowed or a result may beoutputted in S30.

FIG. 4 shows a schematic illustration of a system 10 b to control aworkflow according to an embodiment of the invention. In this case thesystem 10 b is configured as a head-mounted device. This head-mounteddevice can for example be an augmented reality device having see-throughdisplays, on which additional content can be displayed like a display ofa task information 20. Moreover, 10 b comprises an eye-tracking devicehaving eye cameras 22 for capturing images of the eyes of the personwearing the head-mounted device. On the basis of the captured eye imagesgaze direction 24 of the person or also the point of regard 26 can becalculated. Furthermore, the head-mounted device comprises a scenecamera 28, which is configured to capture images of the environment ofthe person. These scene images as well as the eye images can then beprocessed and analyzed by means of a processing unit 30 which can alsobe integrated in the head-mounted device. On the basis of the sceneimages, the processing unit 30 can, for example by means of known objectrecognition techniques, identify certain objects 32 in the environmentof the person. Also, the processing unit 30 can determine additionallyon the basis of the captured eye images and the gaze direction 24 and apoint of regard 26 derived therefrom, whether the person is looking atsuch a certain object 32 or not, especially also how long or when.Furthermore, the processing unit 30 can also determine whether theperson is looking at the task information 20 displayed on thesee-through displays 18 or not. This can be for example done again onthe basis of the captured gaze direction 24 and the known position ofthe displayed task information 20. This system 10 b can for example beused for performing a method described with regard to FIG. 2 or FIG. 3.Based on the captured scene images, the information about the displayedtask information 20 and the eye images the processing unit 30 candetermine whether certain task conditions are fulfilled consisting infor example whether the task has been performed, for example bydetecting that the person has looked at the relevant object 32 and/orconsisting whether the task is allowed to be performed, for example whenit is detected that the person has read the displayed task information20. The head-mounted device can also be configured to output warnings onthe displays 18, e. g. if certain conditions are not fulfilled or thetask had not been performed correctly. The processing unit 30 may alsobe communicatively coupleable to other devices like working tools toprovide control signals, for example to block functions or disable theworking tool, and so on, if certain conditions are not fulfilled, likecertain steps have not been performed correctly by the person.

FIG. 5 shows another schematic illustration of a system 10 b to controla workflow according to another embodiment of the invention. In thiscase the system is not configured as head-mounted device, instead it isintegrated in an inner control panel. This control panel 34 may haveseveral instruments or displays, lamps, and so on. These components ofthe panel 34 are uniformly denoted as 36. The panel 34 may also have acentral monitor 38 for displaying certain information. The task a personP3 now has to perform may be for example scanning all the instruments 36and/or the monitor 38 every thirty seconds. The system 10 c forcontrolling this workflow comprises in this example a remote eye trackerwith a camera 40 and a processing unit 42. The processing unit 42 cananalyze the images captured by the camera 40 and therefrom derive theposition and gaze direction 24 of the person P3. Moreover, the positionsof the respective instruments 36 and the central monitor 38 are alsoknown to the processing unit 42, e. g. saved in a storage device. Theprocessing unit 42 can calculate on the basis of the calculated gazedirection whether the point of regard of the user coincides with therespective positions of the instruments 36 or the central monitor 38. Onthe basis of this information the processing unit can check whether theuser is looking at the instruments and/or the central monitor 38 or not,and even on which of them or also within predefined time intervals. Theprocessing unit 42 can also be coupled to the central monitor 38 toprovide additional task information to the person P3 or to outputwarning information if the person P3 forgets to check all theinstruments 36 regularly.

Therefore, by means of the invention attention aware workflow controlcan be provided that determines tasks and/or steps based on the useracknowledging instructions or warnings and/or the coverage of certainvisual spaces or objects. Also different control models can be providedlike a sequence control, e.g. gaze-based enforcing of workflowsaccording which the next step is only allowed to be executed if acertain gaze pattern was performed, which is the indicator for the useracknowledging an event or instruction or having seen the criticalinformation or done the required steps, as well as completion control,where for example a result or an interim result or a diagnosis or actioncan only be issued or taken if a certain set of visual intakes has beencovered, i. e. if all objects in a piece of luggage of the display of aluggage scanner have been fixated or all areas of certain level ofintensity change in a mammographic x-ray images have been looked at, ora helicopter pilot has scanned all instruments every x seconds.

The invention combines gaze information, visual subject matterinformation with a set of rules to control a process that has one orseveral overarching objectives. These methods and systems involvepreferably eye tracker, a scene camera, a data base with scene data,algorithms to detect scene data in scene video and gaze in scene and onscene data and a set of rules for compliance and control of the visualand actual process. Such a set of rules can also be determined in therelevant scene data based on analysis of a number of performances of theintended procedure by users without skill levels and performanceoutcomes. This can be done analytically, statistically, or via machinelearning approaches using eye tracker data, scene video data and processresult and performance data as inputs to determine the scene data in theset of rules relevant to be observed and controlled for in order toachieve intended process results and performance. This enables toestablish guidelines in order to ensure high quality and to protect aworker or user in dangerous situations or processes. Often guidelinesare not followed out of convenience, however, the here suggested systemand method helps to enforce appropriate visual intakes of the user andpersons to avoid critical omissions in perception or decision-making. Ifonly results of procedures were controlled for, it remains unknown whichvisual input has been missed by the user, leading to an intended resultby chance and thereby having an unstable process or if the intendedresult is not achieved the cause remains hidden. This is even morecritical, when the result can be observed only a long time later or whenit is too late or has become very costly to correct. This is typical e.g. in medical screening and diagnostic procedures or in preventivequality inspection tasks. In these cases, the process itself ensures thequality as the results and their correctness can only be assessed muchlater, often only after irreversible damage has occurred, which shouldhave been prevented by the diagnostic or inspection procedure in thefirst place. However, the invention and its embodiments enable toobserve and monitor each single step and the respective results of aprocedure or process and especially detect when certain steps or tasksare not performed correctly. Therefore, errors or failure can berecognized right away or when the output of incorrect results can beeffectively prevented. Also augmented reality systems can help untrainedpersonal to do tasks which used to require a professional with enforcingcertain sequences and/or coverage of visual intake. This can be realizedmore safely and with less liability. The same applies to the training oftasks where visual perception and feeding of mental is critical for highperformance and good decision-making.

LIST OF REFERENCE SIGNS

-   10 a, 10 b, 10 c system-   12 head mounted device-   12 a scene camera-   12 b eye camera-   14 processing unit-   14 a neural network-   16 storage device-   18 displays-   20 task information-   22 eye camera-   24 gaze direction-   26 point of regard-   28 scene camera-   30 processing unit-   32 object-   34 control panel-   36 instrument-   38 central monitor-   40 camera-   42 processing unit-   P1, P2 test person-   P3 person-   S1, S2 scene data-   E1, E2 eye data-   SL skill level-   RQ result quality-   CP task-specific control parameters

The invention claimed is:
 1. A method comprising: capturing an image ofthe eye of a user; determining a gaze direction of the user based on theimage of the eye of the user; determining, based on the gaze direction,that the user has inspected a plurality of objects in a predefinedorder; and in response to determining that the user has inspected theplurality of objects in the predefined order, enabling a device to allowthe user to perform a task associated with the plurality of objects. 2.The method of claim 1, further comprising: determining a state ofattention of the user based on the image of the eye of the user; anddetermining, based on the state of attention, that the user hasattentively inspected the plurality of objects, wherein the device isenabled in response to determining that the user has attentivelyinspected the plurality of objects.
 3. The method of claim 1, furthercomprising: determining that the user has inspected the plurality ofobjects within a predefined time interval, wherein the device is enabledin response to determining that the user has inspected the plurality ofobjects within the predefined time interval.
 4. The method of claim 1,wherein the plurality of objects includes at least one object in theenvironment of the user.
 5. The method of claim 1, wherein the pluralityof objects includes at least one object in an image presented to theuser.
 6. The method of claim 5, wherein the object in the imagepresented to the user includes task information regarding the task. 7.The method of claim 1, wherein determining that the user has inspectedthe plurality of objects includes comparing the gaze direction withposition information of the certain object.
 8. The method of claim 7,further comprising: capturing an image of an environment of the user;determining, based on the image of the environment of the user, theposition information of the plurality of objects.
 9. The method of claim1, wherein the device includes a working tool.
 10. An apparatuscomprising: a camera configured to capture an image of the eye of auser; a communications interface configured to communicate with adevice; and a processor configured to: determine a gaze direction of theuser based on the image of the eye of the user; determine, based on thegaze direction, that the user has inspected a plurality of objects in apredefined order; and in response to determining that the user hasinspected the plurality of objects in the predefined order, transmit,via the communications interface, a control signal enabling the deviceto allow the user to perform a task associated with the plurality ofobjects.
 11. The apparatus of claim 10, wherein the processor is furtherconfigured to: determine a state of attention of the user based on theimage of the eye of the user; and determine, based on the state ofattention, that the user has attentively inspected the plurality ofobjects, wherein the control signal enabling the device is transmittedin response to determining that the user has attentively inspected theplurality of objects.
 12. The apparatus of claim 10, wherein theplurality of objects includes at least one object in the environment ofthe user.
 13. The apparatus of claim 10, further comprising a display topresent a display image to the user, wherein the plurality of objectsincludes at least one object in the display image.
 14. The apparatus ofclaim 13, wherein the object in the display image includes taskinformation regarding the task.
 15. The apparatus of claim 10, furthercomprising a scene camera to capture an image of an environment of theuser, wherein the processor is further configured to: determine, basedon the image of the environment of the user, position information of theplurality of objects; and determine that the user has inspected thecertain object by comparing the gaze direction with the positioninformation of the plurality of objects.
 16. A non-transitorycomputer-readable medium encoding instructions which, when executed,cause a processor to perform operations comprising: capturing an imageof the eye of a user; determining a gaze direction of the user based onthe image of the eye of the user; determining, based on the gazedirection, that the user has inspected a plurality of objects in apredefined order; and in response to determining that the user hasinspected the plurality of objects in the predefined order, enabling adevice to allow the user to perform a task associated with the pluralityof objects.
 17. The non-transitory computer-readable medium of claim 16,the operations further comprising: determining a state of attention ofthe user based on the image of the eye of the user; and determining,based on the state of attention, that the user has attentively inspectedthe plurality of objects t, wherein the control signal enabling thedevice is transmitted in response to determining that the user hasattentively inspected the plurality of objects.
 18. The non-transitorycomputer-readable medium of claim 16, the operations further comprisingpresenting a display image to the user, wherein the plurality of objectsincludes at least one object in the display image including taskinformation regarding the task.
 19. The non-transitory computer-readablemedium of claim 16, the operations further comprising: capturing animage of an environment of the user; and determining, based on the imageof the environment of the user, position information of the plurality ofobjects, wherein determining that the user has inspected the pluralityof objects includes comparing the gaze direction with the positioninformation of the plurality of objects.